For the production of motion pictures, it is often desirable to combine digitally rendered images with images of a real scene such as may be captured by traditional film cameras or by more modern digital cameras. For example, it may be desirable to digitally create visual effects (VFX) within a filmed scene that would be more expensive or even impossible to achieve using traditional special effects tools. Yet, it is also desirable that the visual effects appear to be completely real so as to preserve the illusion of reality. Methods for rendering digital objects for insertion into photographic images, including what is sometimes referred to as “image-based lighting,” are known in the art. In brief, image-based lighting refers to the practice of gathering real-world information about lighting at a particular point in a photographed or captured scene, and then using that information to render a digital object. The rendered object is thereby illuminated in much the same way as if it were a real object located in the scene, and may be inserted into a photographic image (or a frame of a video sequence) of the scene without appearing out of place.
Currently, several methods for capturing the global lighting of a point in space exists. Current solutions include taking several high-dynamic range pictures of a spherical mirror ball or taking high-dynamic range panoramic pictures with specialized cameras, among others. The simplest way is to use a regular camera to take a photograph of a mirrored ball placed in a scene. A mirrored ball actually reflects the entire world around it, not just the hemisphere toward the camera. Light rays reflecting off the outer circumference of the ball glance toward the camera from the back half of the environment. Another method of obtaining omnidirectional images using a regular camera is to shoot a mosaic of many pictures looking in different directions and combine them using an image stitching program as is known in the art. A good way to cover a particularly large area in each shot is to use a fisheye lens, which enables a user to cover the full field in as few as two images. A final technique is to use a special scanning panoramic camera, such as the PanoCaM™ camera commercially available from SpheronVR AG of Germany, which uses a vertical row of image sensors on a rotating camera head to scan across a 360-degree field of view. However, the specialized cameras and the spherical mirror balls need to be properly characterized and calibrated for taking the light measurements. Furthermore, these systems are adequate for capturing the lighting in only one point in space and one instant in time. However, VFX applications generally require lighting information to be available for an entire region in space, e.g., a set of a motion picture, and, sometimes, for a period of time while the lighting environment changes.
Other approaches use camera arrays to capture the light information. Since, in a camera array, the cameras are evenly positioned on a flat plane, the field of view of the camera array is limited to a certain angle and a certain spatial location. In VFX applications, generally a much larger field of view and light information (virtually at every location) is needed and, therefore, the current camera array solution has limitations for VFX applications.
Furthermore, existing systems do not collect lighting information from multiple locations and/or at multiple times. Existing systems do not systematically build lighting models that allow them to predict or interpolate the lighting in other locations and/or various times. Therefore, the functionality of existing systems for VFX compositing is limited to scenarios where capturing and modeling simple global lighting is sufficient.